System and method for glint reduction

ABSTRACT

Systems and methods for reducing the deleterious effects of specular reflections (e.g., glint) on active illumination systems are disclosed. An example system includes an illuminator or light source configured to illuminate a scene with electromagnetic radiation having a defined polarization orientation. The system also includes a receiver for receiving portions of the electromagnetic radiation reflected or scatter from the scene. Included in the receiver is a polarizer having a polarization axis crossed with the polarization orientation of the emitted electromagnetic radiation. By crossing the polarizer with the polarization of the emitted electromagnetic radiation, the polarizer may filter out glint or specular reflections in the electromagnetic radiation returned from the scene.

CROSS REFERENCE TO RELATED APPLICATIONS

This Application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/573,156, filed on Oct. 16, 2017, which isincorporated by reference herein in its entirety.

TECHNICAL FIELD

This disclosure generally relates to electromagnetic radiation sensorsystems and, more particularly, to active illumination systems.

BACKGROUND

An active illumination system is a system in which an illuminator emitsan electromagnetic signal that is reflected or otherwise returned from ascene of interest. The returned signal is sensed and processed by thesystem to determine useful information about the scene. In activeillumination systems, glints, specular reflections or retro-reflections(such as license plates) often have a higher signal return than surfacesthat scatter light (i.e., Lambertian scatters) due to theirdirectionality of the return. This often poses a problem since thedynamic ranges of the imaging systems are not sufficient to cover boththe bright specular reflections and the less bright scatter reflections.This may lead to either overexposure of the specular reflection andassociated effects (such as blooming on CCD cameras, pixel saturation ina certain area) or underexposure of the scatter returns (and thuspossibly not producing a desired signal-to-noise contrast).

Therefore, there is a need for techniques to reduce glint and theundesirable effects of specular reflections on active illuminationsystems.

DRAWINGS

FIGS. 1A-B are schematic illustrations of an example active illuminationsystem illuminating both specular and scattering object surfaces.

FIG. 2 illustrates a perspective view of an exemplary system forprocessing an image to reduce or eliminate the effects of glint.

FIG. 3 is a schematic block diagram illustrating certain components ofthe imaging system shown in FIG. 2.

FIG. 4 schematically illustrates an exemplary 3D (three-dimensional)imaging system employing at least one of the disclosed techniques formitigating the effect of glint on image capture.

FIG. 5 schematically illustrates another exemplary 3D imaging systememploying at least one of the disclosed techniques for mitigating theeffect of glint on image capture.

FIG. 6 schematically illustrates a further exemplary 3D imaging systememploying at least one of the disclosed techniques for mitigating theeffect of glint on image capture.

FIG. 7 is a schematic diagram of an example 3D system or cameraincluding a modulator and a polarizing grid array and employing at leastone of the disclosed techniques for mitigating the effect of glint onimage capture.

FIG. 8 schematically illustrates another example of a 3D imaging systemincluding a modulator and a polarizing grid array and employing at leastone of the disclosed techniques for mitigating the effect of glint onimage capture.

DETAILED DESCRIPTION

The following detailed description is offered not to limit but only toexemplify and teach embodiments of systems and methods for reducing theeffects of glint in active illumination system. The embodiments areshown and described in sufficient detail to enable those skilled in theart to practice them. Thus, the description may omit certain informationknown to those of skill in the art. The disclosures herein are examplesthat should not be read to unduly limit the scope of any patent claimsthat may eventual be granted based on this application.

The word “exemplary” is used throughout this application to mean“serving as an example, instance, or illustration.” Any system, method,device, technique, feature or the like described herein as “exemplary”is not necessarily to be construed as preferred or advantageous overother features.

As used in this specification and the appended claims, the singularforms “a,” “an,” and “the” include plural referents unless the contentclearly dictates otherwise.

Although any methods and systems similar or equivalent to thosedescribed herein can be used in the practice the invention(s), specificexamples of appropriate systems and methods are described herein.

Also, the use of “or” means “and/or” unless stated otherwise. Similarly,“comprise,” “comprises,” “comprising” “include,” “includes,” and“including” are interchangeable and not intended to be limiting.

The disclosed system(s) and method(s) describe certain techniques forreducing the specular component of a returned signal level in an activeillumination system so that the specular component is comparable to thescatter reflection component. This may increase the dynamic range forboth components and avoids saturation effects that could impact acaptured image and the performance of the system.

FIGS. 1A-B are schematic illustrations of an example active illuminationsystem 10 illuminating both specular (FIG. 1A) and scattering (FIG. 1B)object surfaces 15, 16, respectively. The system 10 includes atransmitter 12 configured to transmit a polarized electromagnetic signal14 for illuminating the surfaces 15, 16. The system 10 also includes areceiver 11 for receiving portions of the electromagnetic signalreflected or scattered 17, 18 from the surfaces 15, 16.

FIGS. 1A-B show two exemplary operational scenarios of the system10—Fig. 1A showing a situation where the system 10 illuminates a highlyreflective surface 15, and FIG. 1B showing a situation where the system10 illuminates a less reflective surface that generally scattersincident light 14 emitted from the system transmitter 12. The receiver11 receives returned portions 17, 18 of the illuminating light 14emitted by the transmitter 12. The light 14 emitted from the transmitter12 may be polarized—linearly, circularly or elliptically.

As shown in FIG. 1A, specular reflections 17, e.g., from mirrored orhighly reflective surfaces, such as surface 15, typically onlydepolarize a small portion of the returning light reflected from thesurface. This is illustrated by the longer arrows in the wave train 17representing the predominate polarization component and the shorterarrows in the reflection wave train 17 representing another, smallerpolarization component. If the incident light 14 is polarized (asillustrated by the single arrows in the incident light wave train 14),this means the returning light 17 is mainly polarized as well with thesame polarized orientation, as shown by the wave train arrows in FIG.1A. This type of reflection is often the case for man-made objects.Objects that produce undesirable glint or specular reflection includethose having highly reflective surfaces, such as mirror or polishedmetal surfaces, corner reflectors, retroreflectors, corner cubes and thelike.

Natural surfaces, e.g., scatter reflection surface 16 as shown in FIG.1B, on the other hand, often do not have a large specular component, andthe returning light 18 may be fully depolarized, as well as scatteredinto a large angle. This is illustrated by the equal-length arrows shownin the returned light wave train 18, which represent polarizationcomponents have similar magnitudes in the returned light.

To reduce or eliminate the glint from the specular surface 15, ahigh-extinction polarizer (not shown in FIGS. 1A-B) may be included inthe receiver 11 of the system 10 (e.g., polarizer 172) that ispositioned orthogonally to the predominate polarized component ofreturning light. By using a transmitter 12 that emits polarized light 14having a known polarization, the polarizer in the receiver 11 can becrossed with the polarization of the emitted polarized light toeliminate or reduce the (polarized) specular component and thus reducethe glint signal level. The specularly reflected light level transmittedthrough such a polarizer may on the same order as normally scatteredlight. Crossed polarizers may be used, that is, one polarizer of thetransmitter 12 or illuminator that emits the light to irradiate theobjects in the scene is at a first polarized orientation, and the otherpolarizer included in the receiver 11 or sensor that detects thereturned portions of the emitted light is at a second polarizedorientation different from the first so that returned specular componentof the received light is reduced or eliminated. The degree to which thepolarizers are crossed with one another can be any suitable value. Insome cases, the axes of polarization of the polarized emitted light andreceiver polarizer are offset from each other by several degrees. Inother cases, there is a high degree of crossing. For example, in someconfigurations the polarizations of the transmitter and receiver may becrossed orthogonally to each other. This may significantly reduce thespecular component returned from the scene. In turn, this may cause thereturned light from the objects to be within the dynamic range of thecamera or sensor included in the receiver 11.

Scenes of interest for the systems disclosed herein may include bothscatter and specular reflection surfaces. Although FIGS. 1A-B show twoseparate scenarios, one having only a specular surface and the otherhave only a scattering surface, the techniques, systems and methodsdisclosed herein can be used in any operational scenarios, includingthose exhibiting both types of surfaces.

FIG. 2 illustrates a perspective view of an exemplary system 104 forprocessing an image to reduce or eliminate the effects of glint(specular reflections from certain objects in a scene). The system 104may be a camera or other imaging system used to capture an image ofscene 100, which includes one or more objects 102. The scene 100 may beirradiated by illumination light 108 emitted from an illuminationsubsystem 110 included in the imaging system 104. Light, both ambientlight and illumination light 108, is reflected or scattered from objects102 in the scene, shown in FIG. 2. Some of the light from the objects102 is received by the imaging system 104, shown as rays 112, and may beincident on a sensor subsystem 120 included in the imaging system 104.

The system 104 includes the illumination subsystem 110, the sensorsubsystem 120, a processor subsystem 140 (shown in FIG. 3), and body 150in which the various subsystems are mounted. The body 150 may furtherinclude a protective cover, not shown. The particular form of system 104may vary depending on the desired performance parameters and intendedapplication. For example, the system 104 may be sufficiently small andlight as to be held by a single hand, similar to a camcorder, and may beconfigured to record relatively close scenes with acceptable resolution.Alternatively, the system 104 may be configured with a larger or smallerform factor.

The imaging system 104 is configured to reduce or eliminate the specularreflections from the objects which may negatively affect the systemperformance. To accomplish this, the system 104 includes an illuminatorthat emits polarized light with a defined polarization. The sensorsubsystem 120 includes a polarizer 172 (FIG. 3) that may be crossedorthogonally with the polarization of the emitted polarized light. Thisconfiguration reduces the glint from specular reflections in the scene100. The emitted light 108 may be a pulse of light or any other suitableelectromagnetic radiation emission or signal having a predefinedpolarization.

Both 2D and 3D imaging systems that reduce or eliminate glint in imagesusing the disclosed methods and systems are described herein. Inaddition, the systems and methods disclosed herein can also be appliedto 1D imaging systems (e.g., line imagers such as barcode scanners).

FIG. 3 is a schematic block diagram illustrating certain components ofthe imaging system 104 shown in FIG. 2. The system 104 may be configuredto capture 1D, 2D or 3D images. Specific examples of certain 3D imagingsystems that employ glint reduction methods are described herein ingreater detail below with reference to other figures. The system 104includes the sensor subsystem 120, the illumination subsystem (e.g.,illuminator) 110, and a processor subsystem 140.

The illuminator 110 includes a light source that is configured toilluminate the scene 100 with a predefined polarized electromagneticsignal, for example, one or more polarized light pulses. The lightpulses may be linearly polarized with a predefined polarizedorientation, for example, a particular axis of polarization.Alternatively, the light pulses may be circularly or ellipticallypolarized in some embodiments.

The sensor subsystem 120 includes a polarizer 172 that is crossed withpolarization of the emitted light pulses from the illuminator 110. Thesensor subsystem 120 also includes a sensor 170 receiving light passedthrough the polarizer 172. The sensor 170 is configured to output one ormore images in response to received light. The processor subsystem 140includes a processor 150 that is configured to process images from thesensor 170 to form a captured image. The processor 150 may do this bycausing the illumination subsystem 110 to emit a light pulse from theilluminator 162. The processor then causes the sensor subsystem 120 (andthe sensor 170 therein) to capture an actively illuminated image of thescene 100, where the actively illuminated image includes portions of thelight pulse reflected or scattered from the scene 100.

The illuminator 110 includes a light source (not shown) and may includetransmission (Tx) optics (not shown), which may include a transmissionlens (not shown) such as a single lens, a compound lens, or acombination of lenses. The illuminator 110 may also include otheroptical elements such as diffusers, beamshapers, and/or the like thataffect characteristics of light emitted by the subsystem 110.

The light source may be any suitable light source, such as one or morelasers, light emitting diodes (LEDs), vertical cavity surface emittinglaser (VCSELs), strobe lights, or the like, but not limited thereto. Theilluminator 110 may be configured to generate one or more light pulses(e.g., laser pulses). Any suitable light pulse can be used. For example,for 3D imaging applications the emitted light pulses may each be aboutor less than 100 ns in duration. E.g., each light pulse may have arelatively short duration such as a duration of 2 nanoseconds or less,for example, between 1 nanosecond and 50 picoseconds.

Other pulse durations may be used depending on the application, such aslonger pulses in the microsecond range. For more traditional imagingapplications, a pulse width of 10 s of microseconds may be used. Forsome applications the pulse duration may be as long as 33 ms (thestandard frame time of a camera operating at 30 frames/second).

Any suitable portion of the electromagnetic spectrum can be used for thelight pulses, for example, a light pulse may be visible light, infrared,ultraviolet radiation, any overlap of these spectrums, or the like.Also, the spectral bandwidth of the light used for the pulses can be anysuitable value, depending on the application. For some imagingapplications, the spectral bandwidth may be a few nanometers to allowfor a spectral filter to be used in the sensor subsystem 120. In someapplications, e.g., indoor usage of the system 104, the spectralbandwidth of the illuminator 162 may be configured so that it does notcoincide or has less overlap with some of the typical output spectrumsof artificial light sources such as fluorescent lights and LED lighting.

The transmission optics may include a Tx lens and/or other opticalelements that are configured to match the divergence of a light pulseemitted from the illuminator 110 to the field of view (FOV) of thesensor subsystem 120. The divergence of a light pulse may be anysuitable value, for example, any angle of 1 degree or greater, forexample, between 1 and 180 degrees, or between 1 and 120 degrees, orbetween 2 and 90 degrees, or between 2 and 40 degrees, or between 5 and40 degrees.

The illuminator 110 emits a light with a predefined polarization. Insome embodiments, the illuminator 110 includes a light source that emitspolarized light, e.g., a laser or laser diode. In other embodiments, theilluminator 110 includes a polarizer (e.g., such as any of the examplepolarizers described herein for polarizer 172) that is crossedorthogonally with the sensor subsystem polarizer 172, for polarizinglight emitted from the light source. In these embodiments, anon-polarized light source may be used. In other embodiments, apolarizer may be used with a polarized or partially polarized lightsource.

The polarizer 172 filters light received from the scene prior to itreaching the sensor 170. The polarizer 172 may be placed at differentlocations along the optical axis of the sensor subsystem 120, e.g., infront of other components or after them, as long as received lightpasses through the polarizer 172 prior to being received at the sensor170.

Any suitable type of polarizer may be used in the illuminator 110 or asthe polarizer 172. The polarizers may be linear, circular or ellipticalpolarizers. For instance, different types of polarizers may be used asthe polarizer 172 to filter the returning light from a scene. Forexample, a linear polarizer transmits only the portion of incident lightthat is projected along its pass axis, regardless of the incidentlight's degree or state of polarization. This portion can be anywherefrom nearly 100% of the incident light to very nearly zero.

Depending on the type of polarizer, the remainder (non-transmittedlight) can be reflected, refracted or absorbed. For example, a plasticsheet polarizer rejects the unwanted component by absorption, andtypically transmits less than 75% even along the pass axis. Wire gridpolarizers reflect and transmit orthogonal linear polarization states,and can work in strongly converging beams across a wide wavelengthrange, but have low extinction ratios especially at shorter wavelengthsapproaching the dimension of the grid spacing. The extinction ratios ofthese polarizers may be around 500:1. Thin film polarizers separate theportions into reflected and transmitted beams, usually with better than98% efficiency, but work well only within a limited spectral and angularrange. Crystal polarizers either reflect or refract the rejectedportion, without significant absorption of either portion, and canachieve extinction ratios on the order of 10⁶:1 over a broad spectralrange, but only over a small range of incident angles. Crystalpolarizers come in many forms, each with unique characteristics. A thinfilm polarizer plate is simple and inexpensive, consisting of a planeparallel glass plate with a coating on one side. It has hightransmittance for P polarization, high power handling capacity and ahigh extinction ratio. The plate is designed for oblique incidence,usually at Brewster's angle. One surface receives a thin film polarizercoating. The transmitted light is laterally displaced by about 0.43times the plate's thickness for glass, but undeviated in direction.

A polarizer with any suitable extinction ratio may be used, for example,an extinction ratio between about 500:1 to on the order of 10⁶:1, forinstance, about 10⁴:1, i.e., ±one order of magnitude.

A thin film polarizing beamsplitter prism may be used as the polarizer172 and offers wider spectral bandwidth than the thin film polarizerplate. The transmitted light is not displaced or deviated. The cubestyle design reflects the S polarized light at 90° to the incoming beam.Deflection angles other than 90°, while somewhat less convenient insystem layout and alignment, offer considerable performance advantages.Prisms with optically contacted or air-gap interfaces achieve muchhigher power handling capabilities than those with cemented interfaces.

The sensor subsystem 120 may include also receiving (Rx) optics (notshown) in addition to the polarizer 172 and image sensor 170. The Rxoptics may include a receiving lens (not shown) that collects reflectedpulse portions from the scene 100. The receiving lens may be anon-collimating lens that focuses the incoming light into an image. Theappropriate aperture size of the lens may depend on the particularapplication, and may be between, for example, 1 cm and 2.5 cm. Otherportions of the reflected or scattered light pulse, e.g., those portionsthat are reflected in directions other than back toward system 104, maynot be captured by receiving optics. Like the transmission lens, thereceiving lens may include a single lens, a compound lens, or acombination of lenses or other reflective or refractive elements.

The Rx optics may also include other optical elements such as one ormore spectral or band pass filters (BPFs), beamsplitters, additionalpolarizers, or the like that affect characteristics of incoming lightreceived by the sensor subsystem 120. In some embodiments, the spectralfilter(s) may be matched to the bandwidth of the pulses emitted from theillumination subsystem 110 such that filter passes light in the pulsebandwidth while blocking light outside the pulse bandwidth.

In other embodiments, Rx optics may also collect broadband or multiband(e.g., visible) information about scene 100, e.g., unfiltered ambientlight that scene 100 scatters or reflects towards receiving optics 172.As such, the receiving lens preferably is configured to reduce oreliminate possible aberrations known in the art of optical system designthat may degrade image quality for one or more of the bands received.

The image sensor 170 creates a plurality of digital images based onlight 112 it receives from the scene 100. The light 112 may includeambient light and returned light pulse portions that that receivingoptics collect. These images contain positional information aboutobjects 102 in scene 100. The image sensor 170 utilizes a focal planearray (FPA) to obtain an image which provides a signal in response tolight illumination that is then digitized. The FPA includes an array oflight-detecting elements, or pixels, positioned at a focal plane of theRx optics that image a scene. Each pixel of the sensor 170 determines anillumination intensity signal that indicates the intensity of lightreceived by the pixel.

The image sensor 170 may be an off-the-shelf CCD or CMOS imaging sensor.In particular, such sensors may be readily commercially available forvisible-wavelength applications, and require no significant modificationfor use in system 104. In one example, image sensor 170 is acommercially purchased CMOS sensor from Sony Corporation havingmegapixel resolution. Some sensors for use in near-infrared applicationsare commercially available, albeit at substantially greater cost thanthe ubiquitous visible-wavelength sensors, and others are currentlybeing developed. It is anticipated that any of a type of optical sensor,including those yet to be invented, may be used successfully with thesystems disclosed herein. Generally, the image sensor 170 includes anarray of pixels, where each pixel can determine the intensity ofreceived light thereon. An image sensor array may include any suitablenumber of pixels, and contemporary sensors often include millions ofpixels. The performance of the image sensor 170 may be characterized bya frame rate, which is how many times the pixel array of the sensor 170may be read per second; and also characterized by a frame time, which isthe amount of time it takes to read the pixel array.

In some embodiments, the image sensor 170 does not include internalstorage and the image data from the pixel array must be read out andprocessed by the processor 150. In other embodiments, the image sensor170 includes on-board memory for storing one or more images captured bythe pixel array so that a prior image does not have to be read-out fromthe sensor 170 before a second image is captured. In a furtherembodiment, the image sensor 170 may include the on-board memory forstoring one or more images captured by the pixel array and a processorfor performing image processing functions typically performed by theprocessor subsystem 140.

The processor subsystem 140 includes processor 150 coupled to a memory160. The processor 150 receives digital image data from the sensorsubsystem 120, and may store the image data in the memory 160 andperform further processing on the image data to remove ambient light andenhance the image of the scene 100. For example, processor subsystem 140may normalize stored images to compensate for variations in reflectanceor scattering between objects 102. Normalization may be particularlyuseful where variations in reflectance or scattering from objects 102are due to active illumination versus ambient illumination. Theprocessor subsystem 140 may also calculate image parameters based on thenormalized images. For example, the processor 150 may be configured toperform digital filtering on image data prior. For example, if ambientlight intensity is low and noisy, filtering out the noise in the ambientand actively illuminated images may improve image quality.

Further, the processor subsystem 140 may process image data thatincludes grayscale or color information about the scene 100. Theprocessor subsystem 140 may further control and coordinate the operationof illumination subsystem 110 and sensor subsystem 120, as describedherein. For example, adjusting the illumination intensity might beuseful.

The functions of the processor subsystem 140 may be implemented inhardware, software, firmware, or any suitable combination thereof. Ifimplemented in software, the functions may be stored as one or moreinstructions or code on a computer-readable medium (e.g., memory 160)and executed by a hardware-based processing unit (e.g., processor 150).Computer-readable media may include any computer-readable storage media,including data storage media, which may be any available media that canbe accessed by one or more computers or one or more processors toretrieve instructions, code and/or data structures for implementation ofthe techniques described in this disclosure. A computer program productmay include a computer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical discstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. Disk and disc, as used herein, includes compactdisc (CD), laser disc, optical disc, digital versatile disc (DVD),floppy disk and blu-ray disc, where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofcomputer-readable media.

The processor 150 may include one or more processors for executinginstructions or code, such as one or more digital signal processors(DSPs), general purpose microprocessors, application specific integratedcircuits (ASICs), field programmable logic arrays (FPGAs), or otherequivalent integrated or discrete logic circuitry. The memory 160 andprocessor 150 may be combined as a single chip. Accordingly, the term“processor,” as used herein may refer to any of the foregoing structuresor any other structure suitable for implementation of the techniquesdescribed herein. In addition, in some aspects, the functionalitydescribed herein may be provided within dedicated hardware and/orsoftware modules. Also, the techniques could be fully implemented in oneor more circuits, including logic circuits and/or logic elements.

FIG. 4 schematically illustrates an exemplary 3D imaging system 500employing the disclosed techniques for mitigating the effect of glint onimage capture. Capturing the 3D position of surfaces and objects in ascene is becoming more and more commonplace for imaging applications.The system 500 can be used in applications such as robotic vision,autonomous vehicles, surveying, and video game controls. The system 500is able to capture the 3D information along with images or video in highresolution in the same way two dimensional (2D) video cameras and cellphone cameras function today. Size, weight, and power requirements forthe system 500 are relevant considerations, and may depend on theapplication in which the system 500 is used.

The system 500, as well as any of the other 3D systems disclosed herein,can be a LIDAR system for measuring distances to objects in a scene byilluminating those objects with a pulsed laser light, and then measuringthe reflected pulses with a sensor. Differences in laser return timescan be used to make digital 3D-representations of the target scene. TheLIDAR embodiment of the system 500 is useful in automotive applications,particularly using the system 500 as a sensor on an autonomous vehicleto detect and sense objects and their positions around the vehicle. Insuch an application, one or more of the systems can be mounted on thevehicle to cover fields of view around the vehicle. The system 500 candetect objects and their positions around the vehicle in real-time asthe vehicle moves along roadways and in traffic.

FIG. 4 schematically illustrates selected components of thethree-dimensional imaging system 500. The operation and functions of thesystem 500 and its components are described in further detail in U.S.Pat. No. 8,471,895 B2, which is incorporated by reference in itsentirety as if fully set forth herein (referred to herein as the “'895patent”). However, the system 500 described here differs from the 3Dimaging systems disclosed in the '895 patent in that it is modified toperform the method(s) disclosed herein for reducing or eliminating glintfrom specular reflections in images, as described below.

It should be appreciated that the functionality of system 500 mayalternatively be provided with other optical arrangements, for exampleas described below with reference to the other figures. As illustratedin FIG. 4, system 500 includes illumination subsystem 510, sensorsubsystem 520, and processor subsystem 540. Each of these subsystemswill now be described in greater detail.

The illumination subsystem 510 emits polarized light, and includes lightsource 511 for generating a light pulse, transmission (Tx) lens 512 forcontrolling the divergence of the generated light pulse, and optionalphase plate or other beamshaping element 513 for enhancing the spatialprofile of the light pulse. The positions of lens 512 and optional phaseplate 513 may alternatively be reversed. These elements may also becombined in a single optic or set of optics. Illumination subsystem 510is in operable communication with controller 541, which may controland/or monitor the emission of light pulses from light source 511, andwhich further may control and/or monitor the divergence thattransmission lens 512 imparts on the generated light pulse. Theillumination subsystem 510 outputs a predefined polarized light signaland may include a polarized light source and/or polarizer (not shown)for polarizing light from a non-polarized light source. In embodimentsof the illumination subsystem 510 that include a polarizer, thepolarizer may be any of those described herein for polarizer 172. Insuch an embodiment, the polarizer may be located along the optical axisof the subsystem 510 in front of the light source 511.

The illumination subsystem 510 preferably generates a light pulse havinga smooth spatial profile, a smooth temporal profile, and a divergence ofbetween, for example, 5 and 40 degrees. The light pulse may be in anysuitable portion of the electromagnetic spectrum, for example, in thevisible band (e.g., 400-700 nm) or in the near-infrared band (e.g., 700nm-2500 nm). Generally, pulses generated in specific regions of thenear-infrared band are considered to be more “eye-safe” than pulses ofcomparable power in the visible band. Light source 511 is configured togenerate a light pulse in the desired electromagnetic band, and lens 512and optional phase plate 513 are configured to provide that light pulsewith the desired divergence and optionally further to enhance thepulse's spatial profile. In some embodiments, light source 511 is alaser producing light pulses having at least 5 μJ energy, or at least100 μJ energy, or at least 1 mJ energy, or at least 10 mJ energy. Suchlaser energies may be relatively eye-safe because of the high divergenceof the laser beam.

A low-coherence laser that may be used as light source 511, as describedin connection with FIGS. 6A-C of the '895 patent, which subject matteris expressly incorporated herein by reference. A low-coherence laser maybe configured to provide high output power or energy for a relativelylow cost, both for pulsed and continuous wave (CW) laser devices. Lowerspatial coherence may also reduce the focusability of the laser on theretina of the eye, thereby improving eye safety. The three-dimensionalimaging system 500 is an example of a wide field-of-view system in whichthe reduced spatial and/or temporal coherence of a laser may be useful.

Illumination subsystem 510 may generate a laser pulse having a largedivergence, e.g., between 1 and 180, or between 1 and 90, or between 1and 40, or between 2 and 40, or between 5 and 40 degrees of divergence,and low spatial and/or temporal coherence, whereas a diffraction-limitedlaser may have a divergence of only a fraction of a degree and a largeamount of spatial and temporal coherence. The large divergence and lackof spatial and/or temporal coherence may reduce the amount of intensityfluctuations in the laser irradiance at the surfaces of objects beingilluminated with the laser beam. The smoother intensity profile of thelaser beam generated by illumination subsystem 510 may improve theperformance of sensor subsystem 520.

In some configurations, a low coherence laser may generate pulses havinga wavelength of 1400 nm or greater, an energy of 40 mJ or greater, and apulse duration of less than 500 picoseconds. There are several gainmedia that emit in this spectral region, including Er:YAG, Cr:YAG, andTm,Ho:YAG. For example, the material Er:YAG has been used to producepulses at 1617 nm having 1 nanosecond pulse lengths and 0.6 mJ output at10 kHz pulse repetition frequencies. However, Er:YAG offers relativelylow gain, making it difficult to scale to higher pulse energies for evenshorter pulse lengths, e.g., 500 picoseconds or shorter. The otherlisted materials may have similar constraints.

Referring again to FIG. 4, transmission (Tx) lens 512 may increase thedivergence of the light pulse generated by light source 511 (e.g., a lowcoherence laser or any other suitable laser, including a high coherencelaser). For example, although the light pulse from light source 511 maybe relatively highly divergent compared to some previously known lasersbecause the pulse contains many spatially and temporally incoherentmodes, the pulse's divergence may in some circumstances still remainwell below 1 degree. Lens 512 may be configured to increase thedivergence of the light pulse to between 5 and 40 degrees, depending onthe distance of the scene from system 500 and the portion thereof to beimaged. Lens 512 may include a single lens, or may include a compoundlens, or may include a plurality of lenses or mirrors, that is/areconfigured to increase the divergence of the pulse to the desireddegree, e.g., to between 1 and 180 degrees, or 1 and 120 degrees, or 1and 90 degrees, or 2 and 90 degrees, or 2 and 40 degrees, 5 and 40degrees, or between 5 and 30 degrees, or between 5 and 20 degrees, orbetween 5 and 10 degrees, or between 10 and 40 degrees, or between 20and 40 degrees, or between 30 and 40 degrees, or between 10 and 30degrees, for example. Divergences larger or smaller may also be used. Insome embodiments, transmission lens 512 may be adjustable, so that auser may vary the divergence of the laser pulse to suit the particularsituation. Such an adjustment may be manual (similar to the manualadjustment of a “zoom” lens), or may be automated. For example,controller 541 may be operably connected to transmission lens 512 so asto automatically control the degree of divergence that lens 512 impartsto the laser pulse. Such automatic control may be responsive to userinput, or may be part of an automated scene-imaging sequence.

Illumination subsystem 510 optionally may further include phase plate513, which is configured to further smooth the spatial profile of thelight pulse generated by light source 511.

It should be noted that although illumination subsystem 510 includeslight source 511, which is substantially monochromatic, it optionallymay include additional types of light sources. For example, illuminationsubsystem 510 may include a white light source for illuminating thescene with white light. Or, for example, illumination subsystem 510 mayinclude other substantially monochromatic light sources in spectralregions different from that emitted by light source 511. For example,where light source 511 generates laser pulses in one particular portionof the visible spectrum, such as in the green region, e.g., 532 nm, suchpulses may cast that hue over the scene. In some circumstances, such asthe filming of a movie, this may be undesirable. Illumination subsystem510 may include one or more additional light sources that generate lightthat, when combined with the light from light source 511, result in theappearance of white light. For example, where light source 511 generatesgreen laser pulses (e.g., 532 nm), illumination subsystem 510 optionallymay further include diodes or lasers or other light sources that emitwavelengths in the red and blue regions, e.g., 620 nm and 470 nm, that,combined with the green laser pulses to produce an illumination thatmaintains the desired scene illumination characteristics. Theillumination system 510 may include the light sources described for thesystem 104.

Still referring to FIG. 4, system 500 further includes the sensorsubsystem 520, which may receive ambient light from a scene along withportions of the light pulse, generated by illumination subsystem 510,that are reflected and/or scattered by objects in the scene. The ambientlight may be visible light from the scene, which light may be fromambient sources as described herein above.

The example sensor subsystem 520 may include polarizer 172, receiving(Rx) lens 521, band-pass filter (BPF) 522, polarizer (Pol.) 523,modulator 524, optional compensator (Cp.) 525, optional imaging lens526, polarizing beamsplitter 527, and first and second FPAs 528, 529.Sensor subsystem optionally further optionally includes white lightimaging subsystem 530, which includes optional dichroic beamsplitter 531and FPA 532. Sensor subsystem 520 is in operable communication withcontroller 541, which may monitor and/or control the operation ofdifferent components of the sensor subsystem 520, such as receiving lens521, modulator 524, optional imaging lens 526, FPAs 528, 529, andoptional FPA 532.

The polarizer 172 is orthogonally crossed with the polarized lightemitted from the illumination subsystem 510. Although shown at the frontof the sensor subsystem 520, the polarizer 172 may be placed elsewherealong the optical axis of the subsystem 520, as long as it is in frontof the sensor FPAs.

The receiving lens 521 may be a non-collimating lens that collects lightfrom the scene and focuses it into an image. The scene may scatterand/or reflect light in a variety of directions other than back towardthe three-dimensional imaging system 500. Some of such light may begenerated by illumination subsystem 510, while other of such light maybe white light or light in a different wavelength range, which may ormay not have been generated by illumination subsystem 510. The amount oflight collected is proportional to the area of the receiving aperture,e.g., is proportional to the area of receiving lens 521.

To enhance the amount of light collected by sensor subsystem 520, thusincreasing the amount of information that ultimately may be contained ineach three-dimensional image, receiving lens 521 is constructed toreceive as much light as practicable for the given application. Forexample, for some applications in which the imaging system is designedto be lightweight and hand-held, with modest resolution requirements,receiving lens 521 may, for example, have a diameter of 1 to 4 inches,or 2 to 3 inches, or for example, about 2 inches, or smaller. Forapplications in which the imaging system is instead designed to providehigh-resolution images for commercial purposes, receiving lens 521 maybe made as large as practicably feasible, for example, having a diameterof 2 to 6 inches, or 2 to 4 inches, or 1 to 3 inches, or, for example, 4inches. The various optical components of sensor subsystem 520preferably are configured so as to avoid clipping or vignetting thelight collected by receiving lens 521 using techniques known in opticaldesign. Additionally, receiving lens 521 and the other opticalcomponents or coatings preferably also have a wide angular acceptance,e.g., of between 1 and 180 degrees, or between 1 and 120 degrees, orbetween 1 and 90 degrees, or between 2 and 40 degrees, or between 5 and40 degrees.

Receiving lens 521 may include a single lens, or may include a compoundlens, or may include a plurality of lenses or mirrors, that is/areconfigured to collect light from the scene and to image the collectedlight into an image plane at a defined position within sensor subsystem520. Receiving lens 521 preferably is configured to reduce or inhibitthe introduction of spherical and chromatic aberrations onto thecollected light. In some embodiments, receiving lens 521 may beadjustable, so that a user may choose to adjust the position of theobject plane of lens 521, or the distance at which the scene is imagedto the defined plan within sensor subsystem 520. In some embodiments,receiving lens 521 can be adjusted to change the angular FOV. Such anadjustment may be manual (similar to the manual adjustment of a “zoom”lens), or may be automated. For example, controller 541 may be operablyconnected to receiving lens 521 so as to automatically control theposition of the object plane of lens 521 or angular FOV of lens 521. Insome embodiments, these adjustments may be performed in part based onthe beam divergence imparted by transmission lens 512 (which also may becontrolled by controller 541). Such automatic control may be responsiveto user input, or may be part of an automated scene-imaging sequence, asdescribed in greater detail below.

Sensor subsystem 520 includes an optional visible imaging subsystem 530,so the light collected by receiving lens 521 is imaged at two imageplanes. Specifically, the collected light passes through dichroicbeamsplitter 531, which is configured to redirect at least a portion ofthe collected visible light onto FPA 532, which is positioned in theimage plane of receiving lens 521. FPA 532 is configured to record acolor or grey-scale image of the scene based on the visible light itreceives. In some embodiments, FPA 532 is substantially identical tofirst and second FPAs 528, 529, and is configured so that the visiblelight image it records is registered with the images that the first andsecond FPAs record. FPA 532 is in operable communication with controller541, which obtains the image from FPA 532 and provides the obtainedimage to storage 542 for storage, which may be accessed by imageconstructor 543 to perform further processing, described in greaterdetail below. It should be appreciated that visible imaging subsystem530 alternatively may be configured to obtain an image based on anyother range of light, for example, any suitable broadband or multibandrange(s) of light.

Light that dichroic beamsplitter 531 does not redirect to FPA 532 isinstead transmitted to band-pass filter 522, which is configured toblock light at wavelengths other than those generated by illuminationsubsystem 510 (e.g., has a bandwidth of ±5 nm, or ±10 nm, or ±25 nm), sothat the remainder of sensor subsystem 520 receives substantially onlythe laser pulse portions generated by illumination subsystem 510 thatthe scene reflects or scatters back towards system 500 and ambientbackground light in the same frequency band. The light transmittedthrough band-pass filter 522 is then transmitted through polarizer 523,which eliminates light of polarization other than a desiredpolarization, e.g., so that the light transmitted therethrough issubstantially all H-polarized, or substantially all V-polarized (orright handed circularly polarized, or left handed circularly polarized).Polarizer 523 may be, for example, a sheet polarizer, or a polarizingbeamsplitter, and preferably is relatively insensitive to angle. Thelight transmitted through polarizer 523 is then transmitted throughmodulator 524, which is positioned at the other image plane of receivinglens 521. The functionality of modulator 524 is described in greaterdetail below. The image plane of receiving lens 521 may be at a locationin sensor subsystem 520 other than in modulator 524.

The modulator 524 optionally may be followed by compensator (Cp.) 525,which may correct phase errors that modulator 524 may impose on the beamdue to variations in the beam angle, thus further enhancing theacceptance angle of modulator 524. Compensator 525 may include amaterial having the opposite birefringence of the material in modulator524. For example, where modulator 524 includes potassium dihydrogenphosphate (KDP), compensator 525 may include magnesium fluoride (MgF₂)which has the opposite birefringence of KDP and is commerciallyavailable. Other materials may be suitable for use in compensator 525,depending on the characteristics of the material used in modulator 524,such as if the modulator material is potassium dideuterium phosphate(KD*P), compensator materials may be rutile, yttrium lithium fluoride(YLF), urea, or yttrium orthovanadate (YVO₄), among others.Additionally, the thickness of compensator 525 may be selected toprovide an appropriate contrast ratio over the acceptance angle of thesystem. For other modulator designs, such as modulator materials thatare oriented such that the crystal axis is orthogonal to the opticalaxis, the compensator may be a second modulator with the crystal axisrotated 90 degrees about the optic axis.

Following transmission through and modulation by modulator 524 andoptional compensator 525, imaging lens 526 images the modulated lightonto first and second FPAs 528, 529. Specifically, polarizingbeamsplitter 527 separates the orthogonal polarization components of themodulated beam (e.g., the H- and V-polarization components, or left- orright-handed circularly polarized components), which it then redirectsor transmits, respectively, to first and second FPAs 528, 529, which arepositioned in the image plane of imaging lens 526. Imaging lens 526 mayinclude a single lens, a compound lens, or a plurality of lenses. Insome configurations, two imaging lens 526 may be placed after thepolarizing beamsplitter 527, with one each in front of FPAs 528, 529.First and second FPAs 528, 529 record images of the modulated lightimaged upon them, and are in operable communication with controller 541,which obtains the recorded images and provides them to storage 542 forstorage and further processing by image constructor 543.

A description of various embodiments of modulator 524 and FPAs 528, 529will now be provided. A description of the calculation of objectpositions and shapes within the scene is provided in the '895 patentwith reference to processor subsystem 540, which subject matter isexpressly incorporated by reference herein. As described in the '895patent, the modulator may be used to vary the polarization of the laserpulse portions reflected from the scene, allowing for the ranges andshapes of objects in the scene to be calculated with high precision. APockels cell or a Kerr cell may in some embodiments be used to performsuch a modulation. However, previously known Pockels cells typicallyhave relatively small apertures (e.g., 1 cm or smaller) and smallacceptance angles (e.g., less than 1 degree) and operate at relativelyhigh voltages, which may make them undesirable for use in imagingsystems. Additionally, the angular extent of the reflected lightreceived by the modulator may be magnified by the inverse of themagnification of the receiving optical elements. As such, it may bedesirable to use a modulator having a wider acceptance angle, a wideraperture, and a lower operating voltage. For example, in thethree-dimensional imaging system illustrated in FIG. 4 the lightcaptured by receiving (Rx) lens 521 may have angles varying between 5and 40 degrees and an aperture of 2-4 inches, for example. Thus, it maybe desirable to provide a polarization modulator having a largeaperture, a low operating voltage, and a large acceptance angle, e.g.,greater than 5 degrees, for example, between 5 and 40 degrees, whileproviding a high contrast ratio, e.g., greater than 300:1, or greaterthan 500:1.

Configurations of the system 500 in which the modulator 524 is a Pockelscell are further described in the −895 patent, which subject matter isexpressly incorporated herein by reference. Although system 500 of FIG.4 is described in the '895 patent as including a Pockels cell-basedmodulator, other types of modulators and/or modulation schemes may beused to encode the TOFs of reflected/scattered pulse portions from thescene as an intensity modulation on an FPA, as is further described inthe '895 patent, which subject matter is also expressly incorporatedherein by reference.

The first and second FPAs 528, 529 are positioned in the focal plane ofimaging lens 526, and respectively receive light of orthogonalpolarizations. For example, polarizing beamsplitter 527 may direct lightof H-polarization onto FPA 528, and may transmit light of V-polarizationonto FPA 529. FPA 528 obtains a first image based on a firstpolarization component, and FPA 529 obtains a second image based on thesecond polarization component. FPAs 528, 529 provide the first andsecond images to processor subsystem 540, e.g., to controller 541, forstorage and further processing, as described in greater detail herein.Preferably, FPAs 528, 529 are registered with one another. Suchregistration may be performed mechanically, or may be performedelectronically (e.g., by image constructor 543).

The FPAs 528, 529 may be off-the-shelf CCD or CMOS imaging sensors. Inparticular, such sensors may be readily commercially available forvisible-wavelength applications, and require no significant modificationfor use in system 500. In one example, FPAs 528, 529 may be commerciallypurchased CCD or CMOS sensors having multi-mega pixel resolution, e.g.,2 Megapixel resolution. Some sensors for use in near-infraredapplications are currently commercially available. It is anticipatedthat any of a variety of sensors, including those yet to be invented,may be used successfully in many embodiments of the present invention.Optional FPA 632 may in some embodiments be the same as FPAs 528, 529.

However, sensors having a particular set of characteristics may in somecircumstances be preferred. For example, as noted above, providing afocal plane array in which each pixel has a deep electron well, e.g.,greater than 100,000 electrons, may enhance the signal to noise ratioobtainable by the system. The focal plane array also, or alternatively,may have a high dynamic range, e.g., greater than 40 dB, or greater than60 dB. Additionally, wells of such effective depths may be obtained bycombining the outputs of pixels of shallower depth (e.g., 4 pixels eachhaving a well depth of 25,000 or more electrons). Preferably, each pixelof the FPA is designed to substantially inhibit “blooming,” so that theelectrons of any pixels that may become saturated do not bleed over intoadjacent pixels.

The processor subsystem 540 includes controller 541, storage 542, imageconstructor 543, GPS unit 544, and power supply 545. Not all of suchcomponents need be present. The functionalities of such components mayalternatively be distributed among other components of system 500,including but not limited to on-board processors on FPAs 528, 529. Asdescribed above, controller 541 may be in operable communication withone or more elements of illumination subsystem 510, such light source511 and transmission (Tx) lens 512, and/or of sensor subsystem 520, suchas receive (Rx) lens 521, optional FPA 532, modulator 524, and first andsecond FPAs 528, 529. For example, modulator 524 may be configured tomodulate the polarization of light pulse portions transmittedtherethrough as a function of time, responsive to a control signal fromcontroller 541. The controller 541 may send a control signal to voltagesource, which applies appropriate voltages to Pockels cells in themodulator 524. Controller 541 is also in operable communication withstorage 542, image constructor 543, optional GPS unit 544, and powersupply 545.

Controller 541 is configured to obtain images from optional FPA 532 andfirst and second FPAs 528, 529 and to provide the images to storage 542for storage. Storage 542 may RAM, ROM, flash memory, a hard drive, flashdrive, or any other suitable storage medium.

The image constructor 543 is configured process the images stored in thestorage 542. Among other things, the image constructor 543 may includeone or more programmable devices, such as a microprocessor or digitalsignal processor (DSP) that are programmed to obtain the stored imagesfrom storage 542 and to construct three-dimensional images basedthereon, as described in greater detail below.

The optional GPS 544 is configured to identify the position and/orattitude of system 500 as it obtains images, and to provide suchinformation to storage 542 to be stored with the corresponding images.Additionally, an accelerometer or other suitable attitude measuringdevice may be used determine an approximate change in attitude of thesystem 500 from one frame to the next in a series of images. Thisinformation may be used as part of a method to register the images to aglobal or relative reference frame. Power supply 545 is configured toprovide power to the other components of processor subsystem 540, aswell as to any powered components of illumination subsystem 510 andsensor subsystem 520.

Responsive to the control signal that controller 541 generates,modulator 524 generates a phase delay between orthogonal polarizationstates for pulse portions transmitted therethrough. This modulation isdescribed in detail in the '895 patent, which subject matter isexpressly incorporated herein by reference. The generated phase delay iswhat permits the system 500 to calculate a TOF and corresponding rangevalue, z, for each pixel in an image, as described in the '895, whichsubject matter is also expressly incorporated herein by reference.

In one configuration of the system 500, first and second discrete FPAs528, 529 and image constructor 543 constitute a means for generating afirst image corresponding to received light pulse portions and a secondimage corresponding to modulated received light pulse portions, whichmay be used to obtain a three-dimensional image based thereon. Forexample, the first image may correspond to the sum of two complementarymodulated images obtained by FPAs 528, 529 (which sum may be computed byimage constructor 543, or alternatively, the sum may be computed byon-board circuitry on one or both of the FPAs), and the second image maycorrespond to the image obtained by FPA 529. In another configuration, asingle FPA and image constructor 543 constitute a means for generating afirst image corresponding to received light pulse portions and a secondimage corresponding to modulated received light pulse portions, whichmay be used to obtain a three-dimensional image based thereon. Forexample, the first image may correspond to the sum of two complementarymodulated images obtained by a single FPA (which sum may be computed byimage constructor 543), and the second image may correspond to one ofthe modulated images. Such configurations may include those in whichmodulators other than a Pockels cell-based modulator were used tomodulate the light pulse portions, e.g., an electro-optic Braggdeflector or other modulator provided herein.

The polarizer 172 crossed with polarized light emitted from theillumination subsystem may be included in other embodiments of the 3Dimaging systems disclosed in the '895 patent, as shown in FIGS. 5 and 6herein. Other than the polarizer 172 and the polarized light from theilluminators, the other components of these systems 1100, 1220 and theiroperation are described in the '895, which subject matter isincorporated herein by reference.

FIG. 7 is a schematic diagram of another example 3D (three-dimensional)system or camera 2010 including a modulator 2014 and a polarizing gridarray 2018 and employing the disclosed techniques for mitigating theeffects of glint on image capture. The camera 2010 also includes thepolarizer 172 that is crossed with the polarization of the light emittedfrom light source 2025. For the present disclosure, the laserillumination (incoming light) 2016 is imaged by the lens 2012 onto thecamera sensor array 2020 through the polarizer array 2018 with a patternof polarization directions or transmission parameters such as shown inFIG. 7. For example, the figure shows alternating horizontal andvertical linear polarizers in array 2018 arranged to be in front of eachpixel 2022, but other arrangements and/or circular or ellipticalpolarization can be used.

For components other than the polarized light source and polarizer 172,the camera 2010 of FIG. 7 and its operation are described in U.S.published patent application 2017/0248796, entitled “3D Imaging Systemand Method,” filed on Feb. 28, 2017, which is incorporated by referencein its entirety as if fully set forth herein (referred to herein as the“'796 application”).

As shown in FIG. 7, the camera 2010 captures 3D information and may alsocapture image or video from a scene 2015 having objects 2017 thatscatter or reflect illumination light emitted from a light source 2025.The light source 2025 may be integrated with the camera 2010 as anillumination subsystem as described in the '895 patent, oralternatively, it may be separated from the camera 2010. The lightsource 2025 may be any suitable means for illuminating the scene 2015with polarized light, including those described in the '895 patent ordescribed herein in connection with FIGS. 2-3.

Although shown as having separated elements in FIG. 7, in someconfigurations of the camera system 2010, the electro-optic module 2021may include the optical modulator 2014, grid 2018, and sensor array2020, as well as an optional polarizer (not shown) located in theoptical path before the modulator 2014 integrally formed together as asingle unit. This highly integrated configuration of the electro-opticmodule 2021 may be constructed using the lithographic, etching anddeposition techniques described herein.

A compact 3D camera system may be achieved by integrating the elementsof a modulated sensor approach described U.S. Pat. No. 8,471,895 B2issued on Jun. 25, 2013, which is incorporated by reference in itsentirety as if fully set forth herein (referred to herein as the “'895patent”) with a polarizing or transmission grid array. Examples of 3Dimaging systems and methods that may be modified to implement themethods and systems described herein are disclosed in the '895 patentat, for example, FIGS. 1-12 and their accompanying written descriptionin the '895 specification. Those portions of the '895 patent describe 3Dimaging systems that can be configured to perform the methods and toinclude the polarizing or transmission grid arrays disclosed in thepresent application, and are specifically incorporated by referenceherein.

Additionally or alternatively, the pulse light source and methodsdescribed in U.S. patent application Ser. No. 14/696,793 filed Apr. 27,2015, entitled “Method and System for Robust and Extended IlluminationWaveforms for Depth Sensing in 3D Imaging” may be used with the systemsand methods disclosed herein, and the subject matter of this applicationis hereby expressly incorporated by reference in its entirety as thoughset forth fully herein.

As disclosed herein, several elements provide the capability of a morecompact, monolithic design either separately or in combination. Insteadof placing complex circuitry and timing algorithms behind eachphotosensitive pixel, the inventive techniques place the requiredtime-dependent elements in front of each pixel or the array of pixels orphoto-sensitive elements. Instead of using electronic means to affectthe voltage or charge signals at each pixel, the inventive techniquesuses optical, electro-optic, or other means of affecting the light fieldin front of each pixel or groups of pixels to affect the photon signal.These optical means may be placed in close proximity to the sensorarray, between the sensor array and corresponding optical elements, orin front of such optical elements to allow extraction of time or depth(e.g., z-axis distance) information from the incident light fieldincluding time-of-flight information.

The use of a modulator (external to the sensor array) as described inthe '895 patent (specifically modulators 524, 700-701 1124, 1224disclosed in the '895 patent, which description is specificallyincorporated by reference herein) to encode the range informationeliminates the need for costly custom sensor array or chip development,especially the challenge of scaling chips that can provide highprecision timing information which have been limited to about 200pixels. Combining the modulator approach with a polarizing grid coupledand aligned to a sensor array eliminates the need to have two separatesensor arrays and bulky polarizing components such as a polarizingbeamsplitter. With a single sensor array, there is alignment andregistration between two virtual arrays. The location of eachpolarization pixel is automatically known relatively to the pixels ofthe orthogonal polarization in position and angle of any surface normal.This reduces manufacturing and calibration complexity.

The use of the polarizing grid also greatly reduces the thickness of theglass or other material that is used for polarization separationelements, which reduces the amount of spherical and other opticalaberrations. In prior systems, these aberrations either degraded theoptical performance of the optical system of the 3D camera, or theoptical system must be adapted with custom designs to remove orcompensate such aberrations. With the techniques disclosed herein, theamount of aberration compensation required of optical elements isreduced or eliminated.

Additionally, the use of the polarizing grid opens the possibility ofmaking the modulator/polarization separation/sensor array into a closelycoupled or monolithic optical assembly that can be used directly withcatalog optical lens or imaging elements. In some circumstances, such aswafer scale manufacturing, no lenses or relay optics would need beplaced between the optical modulator and the sensor array/polarizinggrid. This can reduce the size and cost of the 3D camera system.

The data streams produced and processed by the 3D camera become simplersince there is only one sensor array and no need to time with othersensor arrays. It also becomes simpler to combine multiple 3D cameras ormodules together as described in the '895 patent (for example, to usedifferent range windows and modulation waveforms to extend the rangewindow without worsening the range resolution achievable), such asdescribed in the '895 patent with reference to FIG. 10, which portionsof the '895 patent are specifically incorporated by reference as thoughfully set forth herein.

As shown in FIG. 7, an electro-optic module 2021 includes a grid ofpolarization elements 2018 is placed in front of, or possibly on, thesurface of an imaging sensor 2020 such as a charge coupled device (CCD)or complementary metal oxide semiconductor (CMOS) array of pixels. Insome configurations, the polarization grid layer 2018 can be placeddirectly on the surface of the sensor array 2020 using an additionalstep or steps in the lithographic processing. In others, the grid layer2018 can be placed on a transparent substrate that is then placed on orin front of the sensor array. In other configurations, the polarizinggrid 2018 can be placed within the layers that are above the detector orelectronic sites of a sensor array. The polarizing grid 2018 is alignedsuch that the center of each polarizing element 2019 is positionedapproximately coincident with the center of each pixel 2022. For someconfigurations, the grid 2018 is arranged so that alternating polarizingelements pass orthogonal polarizations. For example, if the firstpolarizing element is oriented to pass vertical polarization, the nextelement in the row or column is oriented to pass horizontalpolarization. Instead of linear polarizing elements, orthogonal circularpolarizing element, both left-handed and right-handed, can also be used.Other configurations may use other patterns of polarizing elements,including elements that pass non-orthogonal polarizations.

Any suitable manufacturing technique may be employed to build thepolarizer element array. For example, the polarizing elements 2018 canbe made using a variety of techniques, including metal wire-gridpolarizers, thin film polarizing layers, stressed polymers, and elementsmade of liquid crystal devices as well as any other technique thatpreferentially passes a particular polarization state over others. Insome cases, the polarizing elements can be made of material that can bechanged with some control signal, either between each pulse or duringthe pulse. Such elements can be deposited by a variety of methods usingfilm deposition techniques. Some can be created by lithographictechniques such as interspersed exposure (including by multiple beams orwavelengths), etch, and deposition steps. Other such elements can becreated by stretching or otherwise stressing materials such as polymers.Some elements can be created by e-beam or laser writing of shapes andstructures of the appropriate spacing or dimensions.

For some configurations, elements that are insensitive to wavelength canbe used to support 3D imagery with multiple illumination wavelengths orwith broadband illumination. In other configurations, elements withnarrow acceptance bandwidths can be used as the polarizing elements tomore effectively discriminate between desired and undesired wavelengthsof light.

By using lithographic fabrication processes, any polarizer grid tosensor array misalignment and non-uniform spacing, non-ideal polarizerperformance, and cross-talk between the pixels can be reduced. Becauseboth the polarizer grid and the sensor array can be fabricated usinglithographic processes, uniformity of spacing are determined by the maskdesign, which is normally accurate to nanometer levels. Alignmentfiducials can be used to align the two grids and lithographic precisionpermits accurately matching the pitch of the grid elements.

Non-ideal polarizer performance would result in location shifts of theminima and maxima of output light. This non-ideal behavior can behandled by calibration of the response at various times. Equally,imperfect polarization contrast (the ratio between the transmission ofthe transmitted polarization and the rejected polarization) can bemanaged by proper system calibration. For example, polarizationcontrasts of approximately 5:1, 10:1, or higher can be used withacceptable performance.

In the event of pixel cross-talk, or light or signal incident on onepolarizer element reaching a pixel other than that corresponding to thepolarizer element can also be accounted for by calibration. Differentcalibrations can be performed to account for any changes in thecross-talk that may occur over short or long time scales. Suchcalibration can be performed at a single time or may be performed atseveral times or during the operation of the 3D camera. Suchcalibrations can be implemented using lookup tables (LUTs) or otherfunctions or forms.

An effect may be performance changes as the angle content of theincident light changes, for example by changing the f/# of thecollecting optics. Higher f/# optics may be used to reduce cross-talk.

Some configurations may reduce cross-talk by constructing the polarizinggrids to use opaque separator bands or structures between pixels. Suchbands or structures reduce the amount of light that can cross from onepixel position to neighboring pixel positions or pixels. In someconfigurations, such bands or structures may also reduce overalleffective transmission efficiency. Other structures can be implementedto reduce cross-talk, including structures on either side of thesubstrate. For example, opaque or reflective structures can be createdin the space between pixels that would block light that is transmittedthrough the grid element from being transmitted to the detector of aneighboring pixel. Such structures or bands may be placed in front ofthe polarizer array, behind the polarizer array, within the layers ofthe sensor array, or around the photosite or photosites of the sensorarray, as well as within the polarizer array itself. In someconfigurations, guard pixels between the polarization states could beused where the signal is ignored. For example, if the sensor array pixelsize is small, for example three microns, a polarizer element might benine microns wide with a three micron separator that covers the guardpixels. Alternatively, guard pixels could be used with no specialseparation existing on the grid structure between elements.

For some configurations, some of the elements of the polarizer array mayhave no polarization properties or reduced polarization properties,forming the basis to determine the normalization signal. Any suitablearrangement of polarization elements and non-polarization elements inthe grid can be used depending on the application and system design.These non-polarization elements can be approximately uniform intransmission for multiple wavelengths or they can vary similar to Bayerpatterns for color cameras or different filters for IR or thermalcameras or other arrangements at other wavelengths or wavelengthregions. For example, they may be opaque or less transmissive of light.

In some arrangements, the polarizer grid elements can be larger than asingle pixel of the sensor array, for example 2×2, 3×3, 4×4, or othermultiple. The elements can also be rectangular, for example, 2×1, 3×2,or other multiple or aspect ratio or any other arrangement that isnon-rectangular in shape. If the grid elements are larger than onepixel, the transmissive elements may be further divided into individualareas that transmit different amounts based on wavelength or angle orother similar optical property.

In the processing software, the detected signal from the pixels in thesensor array 20 can be binned or otherwise processed to improve therobustness of the measurement, reduce sensitivity to noise or otherdeleterious effects, or otherwise improve the signal to noise ratio ofthe individual measurements. Values from different elements or differenttypes of elements can be combined in many ways, depending on thealgorithm implemented and the result desired.

Alternatively, for other modulation schemes, such as Fabry-Perotcavities or other phase-based modulation schemes, where polarizationmodulation is not used, arrays of elements that vary in transmissionbetween elements in some pattern similar to that described above can beemployed instead of polarization elements. Some elements can berelatively low transmission that may provide the needed finesse for aFabry-Perot cavity while some elements can be relatively hightransmission. The high transmission elements (coupled with hightransmission elements on the other side of the Fabry Perot cavity) canbe used to determine the unmodulated reference signal, includinginterpolating the signal to the lower transmission elements fordetermination of the relative modulation signal as described in the basepatent. The arrangement of these pixels can be grouped in various ways,as described in more detail below.

For other configurations, the gain of individual pixels, columns, rows,or other arrangements of groups of pixels in the sensor arrays can beadjusted or set to different values to reduce contrast between thegroups of elements where there is significant signal or to increase thecontrast between pixels or groups of pixels where there is lower signal,thereby increasing the dynamic range of the sensor or 3D camera. Someconfigurations could make use of additional filters that changetransmission in front of pixels or groups of pixels. For example, aBayer pattern RGB filter could be used or other pattern of differingtransmissive properties. Such filter elements could also be used wheremultiple wavelengths of light are used, either for illuminating thescene for the 3D camera or for acquiring specific background or ambientillumination.

An improved way of eliminating the bulky optics that have beenpreviously used in some 3D cameras to separate polarization states is toplace a polarizing element in front of each pixel of a sensor array.Such micro-grid polarizing arrays can be used to measure the absolute orrelative time-of-flight. Absolute distance measurements can be used in a3D camera, for among other things, to reduce error buildup, particularlywhere multiple objects or surfaces are within the scene and where theyare not connected, or the connection is not visible from the camera.

FIG. 8 schematically illustrates another example of a 3D imaging system2120 including the polarizer 172, a modulator 2124 and a polarizing gridarray 2128 and employing the disclosed techniques for mitigating theeffects of glint on image capture. Sensor system 2120 optionally mayinclude visible imaging subsystem 530 show and described in connectionwith FIG. 5 of the '895 patent, which portions of the '895 patent arespecifically incorporated by reference as though set forth in theirentirety herein. The subsystem 530 is omitted from FIG. 8 for clarity.

The system 2120 includes polarizer 172, receiving (Rx) lens 2121,band-pass filter (BPF) 2122, modulator 2124, compensator (Cp.) 2125,optional imaging lens 2126, and FPA 2129, each of which may be the sameas described with respect to the corresponding components illustrated inFIG. 5 of the '895 patent (except for polarizer 172), such descriptionof the FIG. 5 elements of the '895 patent being specificallyincorporated by reference as though fully set forth herein. However,system 2120 also includes polarizer 172 and element array 2128, whichmay be any of the polarizing arrays or transmission-based arraysdescribed, for example, with reference to FIGS. 2-7 of the '796application, which subject matter is incorporated herein by reference.

Some configurations may use all camera elements shown in FIG. 5 of the'895 patent. For example, the system 2120 can include optionalbeamsplitter 2123 which is at any suitable position before the modulator(here, between bandpass filter 2122 and modulator 2124), which directs aportion of the received light to FPA 2119, which obtains an image of thescene based thereon. The remainder of the light is transmitted tomodulator 2124, which modulates the light transmitted there through, andFPA 2129 obtains an image of the scene based thereon. In someconfigurations, the images obtained by FPA 2119 and FPA 2129 may differin that the former is based on unmodulated light, while the latter isbased on modulated light. The image obtained by FPA 2119 may be used tonormalize the image obtained by FPA 2129. Specifically, the intensity atany pixel (i,j) of FPA 2119 may be used as the value I_(total, i,j) inthe distance calculations discussed in the '895 patent with reference toequations (8) to (15), which subject matter is specifically incorporatedby reference as if fully set forth herein. Alternatively, in someconfigurations the intensities measured by FPA 2119 are not needed,instead using the demosaiced intensity sum from FPA 2129 as describedabove.

In other configurations, FPA 2119 is used for images a differentwavelength or wavelengths, such as visible light or infrared light orother spectral region. In other configurations, some of the componentsshown may be omitted or changed in order. For example, in someconfigurations, the beamsplitter 2123 may be replaced by another varietyof polarizing plate or optic or for some instances, omitted altogetherif the incident polarization state is of sufficient quality. In someconfigurations, the compensator 2125 and/or imaging lens can be omitted.The bandpass filter 2122 can also be omitted for suitable environmentswhere background light can be neglected. Alternatively, the components2124 through 2128 or some subset thereof can be repeated in otherconfigurations between beamsplitter 2123 and the FPA 2119. Themodulation patterns between FPA 2119 and 2129 can be the same or ofdifferent lengths or other differences in shape or structure, asdescribed in the '895 patent. The signals obtained from either or bothof the FPAs 2119, 2129 can be combined in algorithms described in the'895 patent.

In other embodiments of sensor 2120, the beamsplitter 2123, imaging lens2126, and FPA 2119 are omitted.

Other techniques described in the '895 patent can be combined with a 3Dcamera using such a transmission array disclosed herein.

It should be understood that, depending on the example, certain acts orevents of any of the methods described herein can be performed in adifferent sequence, may be added, merged, or left out altogether (e.g.,not all described acts or events are necessary for the practice of themethod). Moreover, in certain examples, acts or events may be performedconcurrently rather than sequentially. In addition, while certainaspects of this disclosure are described as being performed by a singlemodule or component for purposes of clarity, it should be understoodthat the glint reduction techniques of this disclosure may be performedby any suitable combination or number of components or modulesassociated with an imaging or sensor system.

The foregoing description is illustrative and not restrictive. Althoughcertain exemplary embodiments have been described, other embodiments,combinations and modifications involving the system(s) and method(s)disclosed will occur readily to those of ordinary skill in the art inview of the foregoing teachings.

1. A method of reducing glint from a returned electromagnetic radiationsignal, comprising: illuminating a scene with an electromagneticradiation signal having a predetermined first polarization; receiving,at a receiver, the returned electromagnetic radiation signal that isscatter or reflected from the scene as a result of illuminating thescene with the electromagnetic radiation signal; and passing thereturned electromagnetic radiation signal through a polarizer includedin the receiver, the polarizer having a second polarization that differsfrom the predetermined first polarization of the electromagneticradiation signal.
 2. The method of claim 1, wherein the polarizer isorthogonally crossed with the predetermined first polarization.
 3. Themethod of claim 1, wherein the polarizer is a plastic sheet polarizer.4. The method of claim 1, wherein the polarizer is a thin filmpolarizer.
 5. The method of claim 1, wherein the polarizer is a crystalpolarizer.
 6. The method of claim 1, wherein the polarizer is selectedfrom the group consisting of a linear polarizer, a circular polarizer,and elliptical polarizer.
 7. The method of claim 1, wherein theelectromagnetic radiation signal is a pulse having a duration of 100 nSor less.
 8. The method of claim 1, further comprising: modulating thereturned portion of the electromagnetic radiation signal as a functionof time; converting into one or more electrical signals the modulatedreturned portion of the electromagnetic radiation signal that has passedthrough the polarizer; and determining 3D information regarding thescene based on the electrical signals.
 9. A system, comprising: anilluminator configured to illuminate a scene with electromagneticradiation having a predetermined first polarization; and a polarizerhaving a second polarization that differs from the predetermined firstpolarization of the electromagnetic radiation, the polarizer configuredto receive a portion of the electromagnetic radiation returned from thescene.
 10. The system of claim 9, wherein the polarizer is orthogonallycrossed with the predetermined first polarization.
 11. The system ofclaim 9, wherein the illuminator includes a light source for emittingpolarized light.
 12. The system of claim 9, the illuminator includes apolarizer configured so that it is crossed with the second polarization.13. The system of claim 9, wherein the electromagnetic radiation is apulse having a duration of 100 nS or less.
 14. The system of claim 9,further comprising: a modulator configured to modulate the returnedportion of electromagnetic radiation as a function of time; an array ofoptical elements receiving the modulated returned portion of theelectromagnetic radiation, wherein at least one of the optical elementshas a predetermined first optical transmission state different from asecond predetermined optical transmission state of another of theoptical elements; and a sensor having an array of pixels correspondingto the array of optical elements, located to receive output from thearray of optical elements.
 15. The system of claim 14, wherein the arrayof optical elements is integrally formed on the array of pixels.
 16. A3D imaging system, comprising: an illuminator configured to illuminate ascene with electromagnetic radiation having a predetermined firstpolarization; a sensor subsystem including: a polarizer having a secondpolarization that differs from the predetermined first polarization ofthe electromagnetic radiation, the polarizer configured to receive aportion of the electromagnetic radiation returned from the scene; amodulator configured to modulate the returned portion of theelectromagnetic radiation as a function of time; and a sensor configuredto receive the returned portion of the electromagnetic radiation thathas passed through the polarizer and modulator; and a processor,operatively coupled to the modulator and sensor, configured to compute3D information regarding the scene based on one or more signals from thesensor.
 17. The system of claim 16, wherein the polarizer isorthogonally crossed with the predetermined first polarization.
 18. Thesystem of claim 16, wherein the illuminator is configured to emit one ormore electromagnetic radiation pulses each having a duration of 100 nSor less.
 19. The system of claim 16, wherein the polarizer has anextinction ratio of about 10⁴:1.
 20. The system of claim 16, wherein thepolarizer is a thin film polarizing beamsplitter prism.